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ABSTRACT 



Student ratings of teaching effectiveness and course quality 
are widely used to evaluate most university courses . This study investigated 
the relationship of medical students' ratings of basic science course 
characteristics to their overall evaluation of these courses. Whether the 
relationship was similar across basic science courses, and whether course 
year has an impact on the instructional characteristics students used in 
making their judgments of overall course quality were also studied. 
Approximately 160 first-year and 160 second-year medical students were asked 
to evaluate basic science courses (14 first-year and 16 second-year) , for a 
total of 3,117 individual evaluations made with an instrument developed for 
the curriculum evaluation process at the medical school . Sixteen 
instructional items were examined as predictor variables. While 10 of the 16 
predictors emerged as significant characteristics using a hierarchical linear 
modeling calculation, the most dominant predictors clearly were students' 
ratings on the administrative aspects of a course and the quality of the 
lectures. Following closely behind these items was the extent to which 
students felt the course promoted active learning. Other variables also 
asserted significant influence, although different predictors emerged for 
each course. These finding suggest that no one student rating item will be 
useful for all purposes, although when they are combined, a sense of overall 
course quality can be obtained. (Contains 5 tables and 15 references.) (SLD) 
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Course Evaluation by Students: The Relationship of 
Instructional Characteristics to Overall Course Quality 

Linda Akel Althouse 
Columbia Assessment Services, Inc. 

Frank T. Stritter, Douglas E. Strong, William D. Mattem 
University of North Carolina School of Medicine 

Introduction 

Student ratings of teaching effectiveness and course quality are widely used to evaluate 
most university courses. Administrators use results of such evaluations in making decisions 
regarding instructors’ promotion, tenure, and salary. Instructors use them in improving their 
course design and instruction. Students use them to reflect on their overall satisfaction with the 
school and the educational experience. Researchers use them to identify effective ways of 
teaching and learning. Because of their widespread use and their importance in decision making, 
student evaluations of teaching effectiveness have been referred to as “the most thoroughly 
studied of all forms of personnel evaluation, and one of the best in terms of being supported by 
empirical research” (Marsh, 1984; Marsh, 1987, p. 369). Extensive research has shown that 
student evaluations are widely used, reliable, stable, and valid (Abrami, d’ Apollonia, & Cohen, 
1990; Feldman, 1988; Marsh, 1987; Murray et al., 1990; Neumann & Neumann, 1983). 
However, most of the literature regarding student evaluation of effective instruction was found to 
focus on teacher behaviors rather than on course characteristics or quality (Abrami, d’ Apollonia, 
& Cohen, 1997). 

In addition, while a number of studies consider the determinants of students’ instructional 
evaluation across widely differing academic areas (Feldman, 1978; Marsh, 1984), no study was 
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found which reviewed instructional evaluation across specific courses within a particular academic 
area, such as medicine. In fact, Abrahams and Friedman (1996) noted that despite the prevalence 
of the use of course evaluations by medical schools, the medical education literature on course 
and curriculum evaluation is sparse. However, in order to better evaluate curricula and provide 
faculty development, medical educators need to know the instructional characteristics that 
contribute the most to students’ perceptions of overall course quality. Therefore, the objective of 
this study was to investigate the relationship of medical students’ ratings of basic science course 
characteristics to their overall evaluation of those courses. The specific questions guiding this 
study were: 

1. What is the relationship of medical students’ ratings of basic science course characteristics to 
their overall evaluation of the quality of those courses? That is, which instructional 
characteristics predict medical students’ overall evaluation of the quality of their basic science 
courses? 

2. Is this relationship similar across basic science courses? 

3 . Does course year have an impact on the instructional characteristics that students use when 

making their judgement of overall course quality? ■' 

Method 

Subjects 

This study was conducted at a large state supported medical school on data collected at 
the end of 1995-1996 academic year. Approximately 160 first and 160 second year medical 
students were asked to participate in a course evaluation at the end of each basic science course. 
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As part of the standard course evaluation practice, 30 courses are rated by a questionnaire 
composed of a common set of seventeen items. The number of students completing the 
questionnaire for each course ranged from 56-152. A total of 3 1 1 7 individual evaluations 
containing the common 17 items were completed. 

Instrument 

The questionnaire was developed by a faculty committee charged to evaluate the 
curriculum and based on issues it felt were important in making decisions about the curriculum. 
The questionnaire consisted of two principal sections. The first section included 16 specific items 
on instructional design which were rated on a 3, 4 or 5 point Likert scale. These items were used 
as the independent variables in our study. The second section consisted of a single question that 
asked the students to rate the overall quality of the course on a 5 -point scale. This item was used 
as the dependent variable. The Cronbach alpha reliability estimate for the instrument was .82. A 
listing of the items and rating choices on the questionnaire is provided in Table 1 . 

Procedure • ^ . * rs ' 

The data for the study were collected as part of the standard curriculum evaluation 
process where students are asked to complete a course evaluation at the end of each of their basic 
science courses. Therefore, no special data collection effort was needed. Once collected, the 
data were analyzed in two stages. In the first analysis, each of the 30 courses was considered 
individually to see which students’ instructional design ratings related to students’ perception of 
overall course quality for each specific course. For each course, the set of 16 instructional design 
ratings was entered into a multiple regression equation. The regression allowed us to determine 
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the amount of variance that these variables explained in students’ rating of overall course quality 
and which of the variables were significant predictors of the overall ratings. Specific attention 
was given to the courses that received the lowest and highest overall rating to determine if the 
same variables emerged as being significant in predicting overall course satisfaction. 

In the second analysis, the data from all the basic science courses were combined. This 
combined data created a hierarchical, multilevel model with students (level 1) nested within 
courses (level 2). As with regression, all models were fit using listwise deletion for missing data. 
Since some items did not apply to some of the courses, these items were left blank when the 
students completed the evaluation. Such missing data values resulted in the loss of three courses 
(courses 1 10, 217, and 221) from the combined data set. Therefore, the combined data included 
only 27 of the 30 courses. Thirteen of these courses were first year course, while the remaining 
14 were second year courses. 

Since the data were collected at the end of each course within the same academic year, 
most students completed an evaluation form on more than one occasion. Therefore, when the 
data were combined the measurements could not be considered independent. However, 
independence of observations is a necessary assumption for regression. Violation of this 
assumption could lead to misestimated standard errors. In fact, the combining of the data yielded 
both course specific (e g. course 1 is easy, but course 2 is difficult) and student-specific (e g. 
student 1 is an easy rater in all courses) error. Therefore, to analyze the effects of the student 
rated variables on ratings of overall course quality on the combined data, hierarchical linear 
modeling (HLM) techniques, which is designed to control for these types of errors, rather than 
multiple regression, were used on the combined dataset. However, since the evaluations were 
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anonymous, student identification was not available making it impossible to correct for student- 
specific error. 

Three series of multilevel (two levels) models were estimated by use of the HLM (Bryk et 
al., 1986) program. For each of these series, level one of the multilevel models was the within- 
unit, student-level model for overall course rating. Level two was the between-unit, course level 
model. 

The first series of models (Model 1) considered only the overall quality ratings for each 
course and no information on the sixteen instructional design items. This series was equivalent to 
conducting a random-effect ANOVA. This multilevel model is commonly referred to as the 
unconditional model and is typically used as a preliminary step in hierarchical data analysis as it 
provides information on the amount of between-course and within-course variation in overall 
course ratings. In addition, it provides information about the reliability of each of the course’s 
sample mean as an estimate of its true population mean. 

The second series of models (Model 2) added the sixteen student rated items, centered by 
the course mean, to level one (within-unit level) of the baseline model. Level two of this model 
was constructed so that the slopes for each of the sixteen items were fixed which meant that the 
effect of each of the items was assumed to be equal for all courses. This model allowed us to 
determine the impact of each of the items on the overall quality course ratings by regressing 
student rating of overall course quality on the sixteen predictor items while controlling for course 
effect. In addition, it allowed us to determine the amount of variation in overall course ratings 
which can be explained by this set of items. 

The third series of models (Model 3) added the course year, first or second, to level two 
(between-unit level) of the model used in the second series. This addition allowed us to determine 
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the effect that the course year had on the ratings given by students. In addition, it allowed us to 
determine the difference in impact that the sixteen instructional items had on overall course quality 
ratings for courses taught in the first year versus the second year. 

Results 

The overall mean ratings and number of completed evaluations for each of the 14 first year 
courses and 16 second year courses are presented Table 2. These means are listed in descending 
order based on their overall quality rating. The average overall mean rating for the first and 
second year courses were 3.556 (sd=.56) and 3.561 (sd=.46), respectively. Since the survey was 
anonymous, we were unable to analyze the differences between respondents and non-respondents 
within each course. However, we found no significant relationship (r=.26) between the number of 
completed evaluations and the overall course quality rating. 



Individual Multiple Regression for the 30 Courses 

The results of the multiple regression for each of the 30 courses are provided in Table 3. 
The amount of variance explained by the 16 predictors in the students’ overall rating of course 
quality ranged from 43% to 86% for each of the 30 courses. There did not appear to be a 
difference in the amount of variance explained between the highest and lowest ranking courses, 
71% versus 68%, respectively. The significant (p< 05) predictors for the lowest ranking class 
included quality of the course syllabus, quality of the large group lectures, and the opportunity for 
interaction with faculty. The significant predictors for the highest-ranking class included 
administrative aspects, integration of subject matter, quality of large group lectures, and the 
extent to which small groups illustrated clinical applications. 
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HIM Analysis of the Combined Data (27 courses j 

Prior to beginning the HLM analysis of the full combined dataset, the correlation between 
each of the indicator variables and the student’s rating of overall quality were examined. The 
linear correlations between these variables and overall course quality rating, as well as descriptive 
data for each item, are presented in Table 4. The magnitudes of the correlations ranged from .003 
to .659. Therefore, no single indicator variable explained more than 43% of the variance of 
another. Also, most of the high correlations involved the dependent variable indicating that a 
strong relationship existed between many of the 1 6 predictors and overall course rating as seen in 
the HLM results. 

The HLM results for the three models using the combined dataset are presented in Table 
5. The preliminary HLM analysis of Model 1 found that prior to controlling for the students’ 
ratings of the instructional characteristics, 25% of the total variance in overall course ratings is 
between courses while 75% of the variance is within courses or at the student level. Therefore, 
most of the variation in overall course ratings is at the student level. However, a substantial and 
significant proportion of the variance is between courses. In fact, a significant difference (x 2= 
804.417, df=26) was found in the mean overall ratings of the courses indicating a group effect and 
confirming the need to perform HLM rather than using a regression analysis. The reliability of the 
estimate of the mean overall course’s sample rating as an estimate of its true population mean 
rating was .96 which indicated that the sample course quality ratings were reliable indicators of 
the true overall rating. 

When the 16 student rated items were added to the level 1 equation to create the second 
multilevel model (Model 2), the following student rated variables were found to significantly (p < 
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.01) relate to overall course quality rating: administrative aspects, subject matter integration, 
quality of course syllabus, fair exams, level of faculty expectation, quality of the lectures, quality 
of the small groups, coordination with other courses, promotion of active learning, and the 
number of classes attended. Items not significantly related included: amount of time spent in 
lectures, amount of time spent in small groups, and the extent to which small groups provided 
opportunities to collaborate with other students, to interact with faculty, illustrate clinical 
applications of the basic sciences, and the extent to which they provided hands-on experience that 
complements the lecture. Highly significant differences (x 2 = 2108.716, df=26) were also found 
among the 27 course means. By comparing this model with the preliminary model (Model 1) , we 
found that the 16 student rated items accounted for 62% {(.784-. 299)/. 784 = 61.9} of the 
student-level variance in overall course ratings. Of the items that were significantly related to 
overall course rating, all were positively related except for the level of faculty expectations as 
evidenced by course exams. If the faculty were viewed as being overally ambitious in their course 
exams, then students tended to rate them poorly. 

In the final HLM model (Model 3), course year was added to the level two between-unit 
model in order to determine the effect course year had on the ratings given by students. As can 
be seen in Table 5, course year was not significantly related to the overall course mean rating (p = 
.970). In fact, course year was only significantly related to two of the student rated items: 
amount of time spent in small groups and whether small groups valued student collaboration. 
When Model 3 is compared to Model 2, we found that the addition of the course year resulted in 
less than one percent of the variation of the student-level variance in overall course ratings. Given 
that the addition of course year did not add to the explanation of variance between or within 
groups, we chose to focus our discussion on Model 2 when discussing the impact of each of the 
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16 predictor variables on overall course rating. However, we did make an exception with the two 
items where course year did make a difference in Model 3. 

Discussion and Educational Implications 

While 10 of the 16 predictors emerged as significant characteristics using the HLM 
calculation, students’ ratings on the administrative aspects of a course and the quality of the 
lectures were clearly the most dominant predictors. The influence of these two characteristics 
was not a surprise as the literature on teacher evaluation frequently cites these two characteristics 
as being influential in assessing the effectiveness of a teacher and/or course (Costin, 1971; 
Feldman, 1997; McKeachie, 1979). In addition, these results agree with the intuitive 
perceptions of curriculum administrators at this institution, as the results may be reflective of the 
structure of the present curriculum that is dominated largely by lectures. Also, administrative 
aspects become very important to students when so much information is presented to them in 
such a short class time. 

Following closely behind these two items was the extent to which students felt the course 
promoted active learning. This result seems to imply that medical students enjoy and want to 
take a more active role in learning material. This awareness supports the research reported on the 
problem based learning methodology which is being introduced across many medical school 
curriculums. However, when lectures are given, the quality of the presentation directly influences 
students' ratings of the overall quality of the course. The combination of these characteristic hints 
to the need and desire for a curriculum that is organized and has high quality lectures with time 
allocated for active, hands-on learning. 
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It is important to note that while the above three characteristics had the most influence on 
perception of course quality, other variables also demonstrated a significant influence attesting to 
the need that all of these items should be considered in the evaluation of a course. In fact, Cashin 
(1997) noted that there are several studies that conclude that student rating forms are 
multidimensional as they measure several aspects of teaching. That is, no one student rating item 
will be useful for all purposes, but combined we can obtain a clearer sense of overall course 
quality. In addition, singular items can be used to better understand what contributes to students' 
perceptions of the quality of a course. 

When individual courses were considered, different predictors emerged for each course. 
This was anticipated since the courses, particularly within the medical school curriculum, have 
widely varying characteristics, which could account for some of the differences found between the 
lowest and highest rated course. The items that were found significant most often across the 30 
courses included the extent that the course engaged the students in active learning (16 of 30), 
quality of the lectures (14 of 30), and administrative aspects of the course (12 of 30). This agreed 
with the three predictor variables found to be influential when the data were combined across all 
the courses. Only one item, amount of time spent in lectures, was found be insi gnifi cant for all 30 
courses. This implied that students did not object to spending time in lectures as long as the 
quality of the lectures was high. While the quality of small groups was found to be significant in 9 
of the 30 courses, the quality of specific group activities was rarely significant (3 or less of 30). 
This result possibly indicated that it did not matter what the small group activities were as long as 
the overall quality of the activities was high. One other interpretation is that the items now asked 
about the small groups, are not the key ones, and that students are determining the overall quality 
of small group activities by other qualities currently not included on the evaluation instrument. 



The year that a course occurred in the curriculum was not significantly related to the 
overall quality rating of a course. In fact, course year was significantly related to only two of the 
predictor variables: time spent in small groups and the extent to which small groups encourage 
collaboration with other students. These variables were stronger predictors for first year students 
indicating that as students progressed through the curriculum, they placed less emphasis on small 
group activities when forming their opinions about overall course quality. However, more 
importantly, course year explained less than one percent of the variation of the student-level 
variance in overall course ratings, indicating a low effect on the determinants of students’ 
perception of overall course quality. This implies that the same evaluation instrument is relevant 
for both first year and second year courses. 

It is necessary to note that while this study addresses the characteristics that students 
perceive to influence overall course quality, a highly rated course does not guarantee a higher 
level of learning. Similarly, we do not know if the same instructional characteristics predict the 
amount of learning that takes place in a course. We also need to emphasize that students' ratings 
of their medical courses do not provide a complete assessment of a course or the quality of 
teaching within the course. For example, the amount of material that is retained by students and 
peer reviews are just two other viable sources of information. Finally, we must take note that 
while the sixteen characteristics were designed to be uniform for all 30 courses, there are many 
differences between each of the courses. For example, it is common within medical school to 
have a course director and then multiple course instructors. However, some courses are taught 
with only one instructor. In addition, as noted earlier, some courses may stress small group 
activities while others function entirely as a lecture course. Some courses may be more problem- 
based while others follow the more traditional lecture modality of teaching. Future research needs 
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to be conducted to determine the effect of varying course structures, in addition to the sixteen 
instructional characteristics considered in this study, on overall course quality within a medical 
school curriculum. Recognizing that the sixteen characteristics do not provide a full 
measurement of course quality, we note that they do provide some useful information. In 
addition, as mentioned earlier, formal student ratings have been shown to provide a reasonable, 
reliable, and valid way of measuring students reactions. 

In summary, certain instructional characteristics of a course, as perceived by medical 
students, do relate to their ratings of overall course quality. While the list of course characteristics 
included in this study may only be a subset of all the variables which can influence students’ 
perceptions of course effectiveness and their learning, the results of this study do add to what is 
understood about effective courses. Faculty interested in increasing students’ opinions of their 
courses should first look at improving the administrative organization of their courses and the 
quality of their lectures. However, faculty must guard itself into completely using the ratings as 
bias free results. For example, some of best, yet toughest, material delivered will sometimes be 
rated low. Yet, it is important that faculty with low ratings critically look at their "problem" areas 
and see what improvements, if any, can be reasonably made. This will take time and commitment, 
but the rewards will be reaped when students regard a course as being of higher quality and, 
perhaps, subsequently, learn more effectively. 
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Table 1 : List of Items and Rating Scales on the Evaluation Questionnaire 



Core Evaluation Questions 

I. Organization * 

Q1 Administrative aspects of the course 

1 - Poor 2 - Fair 3 - Good 4 - Excellent 
Q2 Integration of the subject matter 

1 - Poor 2 - Fair 3 - Good 4 - Excellent 
Q3 Overall quality of the course syllabus 

1 - Poor 2 - Fair 3 - Good 4 - Excellent 

n. Testing 

Q4 Extent to which exams were a fair representation of course subject matter 

1 - Did not seem to agree 2 - Significant Deviations 3 - Only Slight Deviations 4 -Almost Perfect 
Q5 Level of faculty expectations as evidenced by course exams 

1 - Not ambitious enough 2 - Generally realistic 3 -Overally ambitious 

Amount of time spent in large group lectures 

1 - Too Little 2 - Slightly too little 3 - Correct amount of time 4 - Slightly too much 5 - Too much 
Quality of the large group lectures 

1 - Poor 2 - Fair 3 - Good 4 - Excellent 

IV. Small Group Activities 

Q8 Amount of time in small group activities 

1 - Too Little 2 - Slightly too little 3 - Correct amount of time 4 - Slightly too much 5 - Too much 
Q9 Quality of the small group activities 

1 - Poor 2 - Fair 3 - Good 4 - Excellent 

Q10 Extent to which small group activities conveyed the value of collaborating with other students 

1 - Not at all 2 - Modestly 3 - Significantly 4 - Very Significantly 
Q1 1 Extent to which small group activities provided an opportunity to interact with faculty members 

1 - Not at all 2 - Modestly 3 - Significantly 4 - Very Significantly 
Q1 1 Extent to which small group activities illustrated clinical applications of the basic sciences 

1 - Not at all 2 - Modestly 3 - Significantly 4 - Very Significantly 
Q13 Extent to which small group activities provided hands-on experience that complemented lecture material 

1 - Not at all 2 - Modestly 3 - Significantly 4 - Very Significantly 

V. Coordination with Other Courses 

Q14 Coordination of this course with other courses in the curriculum you have encountered 

1 - Poor 2 - Fair 3 - Good 4 - Excellent 

Issues 

Extent to which faculty has set goals for the curriculum to diminish excessive reliance on rote memorization 
and promote more active learning 

1 - Not at all 2 - Modestly 3 - Significantly 4 - Very Significantly 
Proportion of scheduled course activities (lectures, labs) attended 

1 -Less than 50% 2-50-74% 3 -75-90% 4 -More than 90%, but not all 5 -All of them 

VII. Overall Rating 

Q17 The overall quality of the course 

1 - Poor 2 - Fair 3 - Good 4 - Very good 5 - Excellent 



VI. General 
Q15 

Q16 



III. Lectures 
Q6 

Q7 
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Table 2: Descriptive Statistics by Course 





n 


Overall 

Course Quality 
Mean Rating 


sd 


First Year Courses 


Course 101 


133 


4.444 


.783 


Course 102 


119 


4.277 


.780 


Course 103 


149 


3.960 


.936 


Course 104 


150 


3.940 


.647 


Course 105 


81 


3.815 


.838 


Course 106 


90 


3.767 


.984 


Course 107 


129 


3.752 


.884 


Course 108 


97 


3.691 


.939 


Course 109 


125 


3.488 


.768 


Course 110 


55 


3.255 


1.075 


Course 111 


78 


3.167 


1.086 


Course 112 


89 


3.034 


1.027 


Course 113 


123 


2.683 


1.074 


Course 114 


87 


2.517 


.951 


Second Year Courses 


Course 201 


84 


4.321 


.731 


Course 202 


88 


4.284 


.742 


Course 203 


89 


4.112 


.665 


Course 204 


127 


4.016 


.882 


Course 205 


78 


3.974 


.789 


Course 206 


102 


3.647 


.940 


Course 207 


74 


3.595 


.859 


Course 208 


80 


3.438 


.953 


Course 209 


135 


3.400 


.899 


Course 210 


81 


3.395 


1.008 


Course 211 


140 


3.329 


1.021 


Course 212 


96 


3.260 


.886 


Course 213 


76 


3.237 


.798 


Course 2 14 


58 


3.190 


.888 


Course 215 


83 


3.096 


.958 


Course 216 


98 


2.684 


.892 
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Table 3: Regression Coefficients of the 16 Questionnaire Items by Course 



£ 

a 



O' 



o 



o 



o 



o 



o 



o 



00 

o 



r-, 

o 



8 



O' 



S' 



O' 



cs, 

o 



o 



^ - -- ■- no no no oo oo r- no no 



r*- r- no vs vo no 



O n (N - » 



O O C*S O On On l — ‘ © rsJ <~sj f*-j »-S in 

h\q^^^oohopPoe8SPP!oh 



^ ^ C ’^OOOO — Og^cSOC CCV)0Cno\-hJ!n-r n--n 

oo»-^-^-ppq^-ooo^-^- oofo^*— *h^-Uioo — oi-o*- 



ics o 

~-©©©eSt'-V5VO©00f*JVSicscS 

“ “ (S - - 



NO 

— On On _ ... _ _ _ 

^0»Ni-0-(SNfS 



vooNfnoenr'N^x^eoNn^Troo 

^^5^°°!: jocTf©^SS--© 



fsninttoo'n^vOhh-ocrr 

©~-©fS^-^-©^-^-~ -©©©~- 



on (N ■ — csnd — *csTrics©rso©r--eso 

<— ;^<Np^-p 0 00fS^^-00 _ 



m ^r 
r- vs 
© ^ 



^O^tnr, rs^xochno 

nh"hOC(N^CITOOC 

o © o — o — © <s — — 



TT 00 

On OS 

O N 



© O cs vs tj- 

N » »n 0\ O 

<S (N o O (N 



Nf n oo I — i/~> co o 

0 0 0 ^-0 c © 



vsrsr--icsics©r'*'SNONO 

SOVjTj-OC-in^Cm 

— oooo 



f*> C*S C*s to r*s 

Vi — <N ~ 

cs © © © — 



ooTrr-ooo©©©oo — nooooi-- 
f*svsf*socvsvso©m©©Trt'smoo 
— | p p — < o p o — © <n o o <n cs 



2?2?3^? C( noo-onom^ 
50 \OyOh<N(NOr)'tvOO , n»r) 
oooo<so — O © CS © © ~ f*l 



2S£^p^9?^? Tr ° iy ^ r " OOT: J" T: J"^- « I s - 



3oS82S:5S25^3^ 



r'-NOOrsmvsc'STrr-'Tf>r--©vs© 

oo^oth^Troo^^roojhc 
© © *— — o©©r^ooo(N 



MjwrroNro , M^oMMTr«n « 
-ON\oojin(S<np-^(snTr^ON>N 
o — «— -oo> — ' o o o . — ©•— ©© C — 



^ ^ t CS CS © 

cs vs v> tt r~~- r~~ 

— O O O CS © 



•n tJ- © SO C- CS 
", Tt - o »n C ON 
5 *— oo^r^-^-© 



©ly^f^r-vorscNcsocrs^-TTV^^ 1 ^ 

qo~ oscsrs — *— r'sooo^-r's c ro 



hhvcinhoO'-fNh-oJh--- 
NOrs^romTrmvorscNOTrmr^ 
0©~*0~-©000©~-~- — © 



o— <000^fS0^N*^0 — ©coo 



^(SwSOnOvnmNMt^OOvOO 
^a\(N^X»0^hlO»\0-h 
~-cs~-rsocs • o o © cs 



vs — 

r- © 

cs — 



rs — 
© © 
© r- 



IT) CS 

** CS 

N — 



c © 
m no g 



CS NO 

© NO 



CS CS 
>A X « 
O fS (N 



no ~ cs © © r-~ t — t^- tj- © oo o vs 
^©Omi— 

00^00000000 — — — , 



o r- 

— © 



NO TT 
r- m 
o o 



r- o 

Tf © 

o © 



NO 00 vO 
vs — o 
o © © 



© tt 

r~- so 

o o 



m vs © 

— © — 

0 — 0 



© 
o — 
o — 



— NO 
vs © 

© o 



t"" ' — ^ I NO r*j f- © © 00 CS 

i^i o m x c h vc to h 

O — O © — — O — © © — 



Tj- © 

m cs 

— © 



CS TT 

^r 
© 



* s 



00 

00 

o 



00 

© 



•D © 
l> OS 



^ tt 

OC Tj- NO 
— — © 



VC^fShinhoCNTfS. — oc 
ir-ntnr,ONXn©fsoc 
*-H© 0 © 0 ^T— — O — O 



CS O 00 
© co vs 
— © — 



rr no 

— CS 



CS f*i 

vi cs 

— © 



VC 

VC 



CS m 

cs r- 

— © 



CS — 
NO 00 
© © 



© NO NO I"" VI CS © NO e*S © © © 

— r*s vs cs — ©Ncr^in©(N©NO© 

~!~-© — — ~*~*00 — ©CSO — 



ovi^Tr(N^^<stnxxTr-h 
a©\0M-nv3hoc © tt cs r- 

' CM © 



— — — — CS O O O 



n M 

i/s vs m 

cs — 



S NO © 
cs oo 
© © © 



S $ 

•H fS 



© r- 

(N © 
— © 



TT (N 

— r- 

© <s 



m (N 
— r- 
© — 



«o X NO 

© — 

© r> — 



(N rn 

© 

fO — 



00 © 
— <N 
CS © 



00 © 
© ics 

© — 



© — 
tt ^r 
cs — 



t+i © © m 

© 00 ^ ICS 

^ © © © rs 



©^-©©© ©©^© 

rjr^Tr©©r-Troe — 

<N — — — © — fS — 



— © © 
»o — 
(S (S o 



» — 

l> ics 



VC rn 

CS — 



V) 00 o 
, *r © © 

*H •— © 



2 ?J 



— (N 

— vc 

CS 



oo c*5 
© oo 
cs cs 



^ © CS © NO NO © 

(N - t Ci h h M 



S O © CS CS I s - 00 

(Nth' ~ ' 



oc © r^ 



ICS © © 

r** r- oo 



ICS NTS 
— NO 



oc r^ 

00 NO 



ICS © — 

no cs r- 



cs cs 

© NO 



tt 

ICS ICS 



3 



3 s 

U 

CQ e> 

>- 3 
£ (J 



cs n ^ m o 

© © © © © © 

888 8 5 «" 



3 



uuuuuuuuuu 




Tf 


vs 


\D 




00 


© 


© 




CS 


m 


Tj- 


vs 


© 


© 


© 


© 




© 


© 




—m 












CS 


cs 


CS 


rs 


CS 


CS 


CS 


CS 


CS 


<N 


CS 


CS 


CS 


« 


Q£ 






a 


& 




u 






o 








£ 


£ 


£ 


£ 


£ 


£ 


£ 


£ 


£ 


£ 


£ 


£ 


s 


Z3 




3 


s 




3 


9 


3 


3 


3 


D 


3 


o 


o 


o 


O 


o 


o 


O 


o 


O 


o 


O 


o 


o 


U 


U 


U 


U 


U 


U 


U 


U 


U 


U 


U 


U 


U 



•ri 

o 

II 

S 8 



5 B 

it 

w O 
2* 
I 

cq Qj 

>, -h 

«■= « 



CO 



CQ 



>- 

CL. 

O 

CJ) 

J — 
OO 
LU 
CQ 



ERIC 



Table 4: Correlations and Descriptive Statistics of the 16 Questionnaire Items and Overall Course Rating 
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Table 5: HLM Results for Within and Between Groups Effects 





Model 1 
Effect se 


p Effect 


Model 2 
se 


P 


Effect 


Model 3 
se 


p 


Fixed Effects - Within-Course 
















Overall Mean Rating 


2.447 .101 


.000 2.448 


.101 


.000 


2.461 


.328 


.000 


Administrative Aspects 




.239 


.023 


.000 


.198 


.069 


.009 


Subject Matter Integration 




.165 


.021 


.000 


.119 


.064 


.076 


Quality of Course Syllabus 




.154 


.018 


.000 


.128 


.055 


.030 


Fair Exam Representation 




.129 


.020 


.000 


.110 


.062 


.089 


Level of Faculty Expectation 




-.155 


.027 


.000 


-.131 


.083 


.126 


Time in Lectures 




-.014 


.158 


.398 


.049 


.049 


.322 


Quality of Lectures 




.227 


.022 


.000 


.206 


.068 


.006 


Time in Small Groups 




.007 


.015 


.660 


-.101 


.045 


.033 


Quality of Small Groups 




.108 


.019 


.000 


.093 


.059 


.130 


Small Group - Collaboration 




.006 


.017 


.739 


-.108 


.053 


.052 


Small Group - Faculty Interaction 




.026 


.021 


.214 


-.022 


.062 


.722 


Small Group - Clinical Applications 




.030 


.021 


.157 


.121 


.062 


.061 


Small Group - Hands-on Experience 




.024 


.019 


.223 


.010 


.059 


.863 


Coordination with Other Courses 




.107 


.020 


.000 


.112 


.063 


.085 


Active Learning 




.196 


.017 


.000 


.231 


.052 


.000 


Proportion of Classes Attended 




.108 


.014 


.000 


.042 


.044 


.355 


Fixed Effects of Course Year on Level- 1 
















Variables 
















Overall Mean Rating 










-.008 


.205 


.970 


Administrative Aspects 










.031 


.046 


.514 


Subject Matter Integration 










.027 


.044 


.545 


Quality of Course Syllabus 










.017 


.036 


.646 


Fair Exam Representation 










.013 


.039 


.744 


Level of Faculty Expectation 










-.017 


.054 


.759 


Time in Lectures 










-.044 


.032 


.172 


Quality of Lectures 










.014 


.044 


.746 


Time in Small Groups 










.073 


.030 


.022 


Quality of Small Groups 










.010 


.039 


.798 


Small Group - Collaboration 










.072 


.034 


.045 


Small Group - Faculty Interaction 










.036 


.043 


.413 


Small Group - Clinical Applications 










-.067 


.043 


.134 


Small Group - Hands-on Experience 










-.016 


.040 


.695 


Coordination with Other Courses 










-.003 


.040 


.942 


Active Learning 










-.024 


■P33 


.480 


Proportion of Classes Attended 










.043 


.027 


.126 



Between Course Variation 


Est. 

.264 


Model 1 
X 2 df 

804.417 26 


p Est. 

.000 .269 


Model 2 

X 2 ^ 

2108.716 26 


p Est 

.000 .280 


Model 3 
X 2 df p 

2121.125 25 .000 


Within Course Variation 


.784 




.299 




.298 




Reliability of Estimate of 
Overall Course Mean Rating 


.961 




.985 




.986 




Percent of Variance 
Explained by 16 Predictors 






61.9 




62.2 
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